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1.  INTRODUCTION 


Certain  tactical  problems  take  the  form  of  inquiries  into  the  best  way  of 
getting  from  A  to  B,  where  A  and  B  are  positions  in  a  continuous  state  space. 
The  optimal  route  from  A  to  B  is  not  necessarily  a  straight  line:  ocean 
currents  or  winds  may  cause  a  ship  to  be  routed  indirectly  to  take  advantage 
of  favorable  areas,  or  certain  regions  may  be  threatening  (typhoons,  enemy 
units)  or  even  non-feasible  (land).  A  "route"  being  a  complicated 
mathematical  object,  it  should  be  expected  that  the  time  required  for 
computation  of  an  optimal  route  will  be  significant,  and  that  it  will  be 
sensitive  to  the  way  in  which  the  optimal  routing  problem  is  formulated  and 
solved.  This  technical  report  describes  a  somewhat  unconventional  approach 
to  formulation  and  solution.  It  includes  a  program  demonstrating  the 
technique  in  a  problem  where  a  submarine  is  to  be  routed  past  several 
listeners  trying  to  detect  it. 


2.  DISCRETE  DYNAMIC  PROGRAMMING  VERSUS  CONTROL  THEORY 


We  will  take  the  objective  to  be 

T 

minimize  J  f(x(t),z(t),t)dt 


(1) 


subject  to  some  constraints  on  the  control  variables  z(t).  x(t)  for  0  <  t  <  T  is  the 
route  to  be  optimized,  generally  a  vector.  For  brevity  the  arguments  of  x  and 
z  will  be  omitted  below,  but  each  is  nonetheless  a  function  of  time  (t).  f(x,z,t) 
is  the  rate  at  which  penalty  is  accrued  at  time  t.  For  example  f(x,z,t)  might  be 
fuel  used  per  unif  time  if  the  problem  were  to  minimize  fuel  usage  or  rate  of 
being  hit  by  AAA  if  the  problem  were  to  navigate  an  aircraft  through  defenses 
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with  as  little  risk  as  possible.  The  control  variables  z  influence  the  state  x 
through  the  equation 

dx/dt  =  g(x,z,t),  (2) 

where  the  function  g  (like  the  function  0  is  given.  The  simplest  case  is  where 
g(x,z,t)  =  z,  in  which  case  z  can  be  interpreted  as  the  controllable  velocity 
corresponding  to  x.  This  case  is  also  the  part  of  Control  Theory  that 
corresponds  to  the  Calculus  of  Variations. 

The  important  feature  of  (1)  is  that  it  is  an  accumulation  over  time.  This 
fact  is  exploited  in  both  discrete  Dynamic  Programming  (DDP)  and  Control 
Theory,  the  two  commonly  applied  methods  of  solution.  In  DDP,  the  state 
space  is  made  discrete  and  the  minimization  is  performed  by  exhaustion;  DDP 
owes  its  power  to  a  clever  ordering  of  the  required  minimizations,  rather 
than  to  any  exploitation  of  analytic  properties  of  f  or  g.  DDP  produces  global 
optima,  an  advantage,  but  it  suffers  from  Bellman's  curse  of  dimensionality — 
the  number  of  state  variables  and  the  coarseness  with  which  each  is  measured 
must  be  carefully  controlled  lest  solution  times  rapidly  become  large. 

Control  Theory  does  not  require  that  state  variables  be  measured  coarsely. 
In  fact,  there  would  be  nothing  illogical  about  using  double  precision 
arithmetic  in  manipulating  them.  Control  Theory  suffers  instead  from 
having  to  solve  two-point-boundary-value  problems.  Roughly  speaking,  in 
applying  Control  Theory  one  easily  obtains  an  optimal  solution,  only  to 
discover  that  the  wrong  problem  has  been  solved.  The  main  computational 
effort  comes  in  manipulating  the  wrong  problem  into  the  right  one.  Even 
when  an  optimal  solution  to  the  right  problem  is  obtained,  it  may  only  be  a 
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local  optimum.  Control  Theory  thus  has  its  own  difficulties,  different  from 
those  of  DDP  but  equally  serious. 

It  should  be  mentioned  that  Dynamic  Programming  is  actually  the  more 
general  af  the  two  techniques,  since  the  functional  equation  of  (continuous) 
DP  can  be  used  to  derive  the  necessary  conditions  of  Control  Theory  (Bellman 
and  Dreyfuss  [1962],  Jacobson  and  Mayne  [1970]).  Bellman's  Principle  of 
Optimality  appears  to  be  the  fundamental  observation:  if  the  optimal  route 
from  A  to  B  passes  through  C,  then  the  parts  from  A  to  C  and  from  C  to  B 
must  also  be  optimal  for  their  respective  problems. 

The  gist  of  the  preceding  paragraphs  is  that  one  can  either  begin  by 
imposing  some  sort  of  grid  on  the  state  space,  in  which  case  DDP  is  the 
natural  optimization  technique,  or  one  can  begin  by  attempting  to  exploit 
analytic  properties  of  f  and  g,  in  which  case  the  necessary  conditions  of 
Control  Theory  are  the  natural  result.  If  one  chooses  the  latter,  one  must  be 
prepared  for  the  possibility  that  optima  may  be  local,  rather  than  global. 

Most  current  tactical  decision  aids  employ  DDP  in  route  optimization. 
Klapp  [1979]  describes  the  approach  of  the  Fleet  Numerical  Oceanography 
Center  in  routing  ships,  essentially  the  imposition  of  a  network  of  grid  points 
on  the  ocean.  The  Naval  Oceanographic  and  Atmospheric  Research 
Laboratory  also  plans  to  incorporate  a  tactical  environmental  ship  routing 
(TESR)  function  as  part  of  the  Tactical  Environmental  Support  System. 
Weissinger  [1987],  describes  an  approach  to  TESR  wherein  states  are  positions 
in  space-time.  He  estimates  computation  times  from  .25  hours  to  15.8  hours, 
depending  on  how  coarse  the  grid  is,  on  a  HP9020  microcomputer.  The  U.S. 
Air  Force  also  incorporates  some  path  optimization  within  its  Mission 
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Planning  System  software.  Jones  [1986]  describes  how  the  WARPATH 
algorithm  computes  an  optimal  route,  and  discusses  alternatives.  All  of  these 
tactical  route  optimization  programs  find  the  shortest  route  through  a 
network,  with  the  node  and  arc  costs  depending  on  the  application  (Deo  and 
Pang  [1984]).  The  coarseness  of  the  network  is  a  crucial  consideration  in 
determining  run  times. 

The  coarseness  inherent  in  the  DDP  approach  suggests  that  some  tactical 
routing  problems  might  be  better  founded  on  Control  Theory.  Construction 
of  such  a  prototype  is  the  goal  of  the  rest  of  this  report.  The  technique  is 
somewhat  novel  and  fully  described  in  the  Appendix.  The  main  idea  is  to 
eliminate  the  two-point-boundary-value  problem,  even  at  the  cost  of 
temporarily  producing  solutions  that  are  non-optimal.  The  technique  is 
roughly  steepest-descent — a  given  route  is  gradually  warped  into  something 
optimal  by  making  first-order  corrections,  with  the  current  route  being  at  all 
times  feasible.  The  initial  route  is  a  user  input.  The  technique  is  employed  to 
solve  a  problem  where  a  submarine  attempts  to  go  from  A  to  B  without  being 
detected,  the  objective  being  to  minimize  total  radiated  energy  received  by 
enemy  listeners.  The  application  and  the  prototype  software  are  described  in 
the  next  section. 

As  mentioned  above,  one  of  the  weaknesses  of  Control  Theory  is  that 
optima  may  be  local,  rather  than  global.  This  is  of  particular  concern  if  there 
is  reason  to  believe  that  the  tactical  problem  is  likely  to  have  multiple  optima. 
This  will  be  the  case,  for  example,  in  problems  where  a  vehicle  is  to  be  routed 
past  obstacles,  since  an  "obstacle  on  the  left"  path  cannot  be  warped  into  an 
"obstacle  on  the  right"  path  without  passing  over  the  obstacle.  The  more 
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obstacles,  the  more  local  optima  must  be  expected.  In  such  problems,  Control 
Theory  will  be  a  useful  approach  only  if  there  is  some  other  reliable 
mechanism  for  selecting  a  good  starting  point;  i.e.,  a  starting  route  that  is 
likely  to  lead  to  a  route  that  is  globally  as  well  as  locally  optimal.  In  two 
dimensional  problems  this  mechanism  may  very  well  be  the  human  user  of 
the  decision  aid,  since  humans  are  good  at  seeing  relationships  in  two 
dimensions.  The  division  of  effort  would  be  that  the  human  deals  with 
topological  issues,  while  the  decision  aid  (via  Control  Theory)  deals  with 
detailed  questions  about  direction  and  possibly  speed.  This  is  the  approach 
taken  in  JITTER,  the  program  to  be  described  in  the  next  section.  The 
problem  is  first  presented  graphically  to  the  user  in  such  a  manner  that  a 
reasonable  route  can  be  selected.  JITTER  then  assumes  a  starting  point  where 
the  speed  along  the  inprt  route  is  constant,  eventually  warping  it  into 
something  that  is  locally  optimal,  possibly  with  variable  speed. 

3.  SUBMARINE  TRANSITS  AND  THE  JITTER  PROGRAM 

Submarines  radiate  acoustic  noise,  with  the  amount  of  noise  power 
radiated  being  a  strong  function  of  submarine  speed.  In  trying  to  get  from  A 
to  B  without  being  detected  by  an  enemy  listener  at  C,  a  submarine  may  be 
tempted  to  steer  far  away  from  C,  but  in  doing  so  may  be  forced  to  go  so  fast 
(‘ime  being  constrained)  that  a  remote  detection  by  C  may  occur  anyway.  If 
there  are  actually  several  enemy  listeners,  the  best  track  can  be  expected  to  be 
sinuous,  but  not  so  sinuous  that  its  length  forces  extremely  high  speeds. 
Determination  of  the  best  track  in  these  circumstances  is  a  rather  subtle 
problem,  one  that  might  reasonably  be  aided  by  a  computer.  This  is  the  goal 
of  the  prototype  program  JITTER. 
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We  will  take  the  objective  to  be  minimization  of  the  total  energy  received 
by  all  listeners.  It  could  be  reasonably  argued  that  a  better  criterion  would  be 
minimization  of  the  maximum  power  received  at  any  time  by  any  listener. 
The  truth  is  somewhere  in  between  (Boyd  [1989]),  but  there  is  little  choice  of 
criterion  if  Control  Theory  is  to  be  easily  applied — the  criterion  must  be  total 
received  energy. 

If  there  are  n  listeners,  the  power  received  by  all  of  them  at  time  t  will  be 
taken  to  be 


P(0  =  £{*.  /  rz (A)2 }{l  +  «v(f )4 }{l - y3 cos(2^ (0)},  (3) 

i=i 

where  (see  Figure  1): 

Tj(t)  =  distance  to  listener  i 
v(t)  =  submarine  speed 

8j(t)  =  bearing  to  listener  i,  relative  to  the  submarine's  bow 


Figure  1.  A  Submarine  Passing  through  Three  Listeners 
The  three  factors  in  {  }  in  (3)  can  be  thought  of  as  basic  radiated  power 
followed  by  two  correction  factors.  Division  by  rj(t)2  in  the  first  factor 
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represents  the  assumption  of  spherical  spreading;  effects  of  reflections  and 
refraction  are  being  ignored.  The  constant  S;  will  depend  on  i  if  some  listeners 
are  more  sensitive  than  others.  The  power  required  to  drive  a  submarine 
increases  roughly  with  the  cube  of  speed.  A  small  but  increasing  fraction  of 
this  is  radiated  as  speed  increases,  so  v(t)  is  raised  to  the  fourth  power  in  the 
speed  correction  factor  (an  ulterior  motive  here  is  that  raising  something  to 
the  fourth  power  doesn't  require  an  exponentiation).  The  parameter  a  is  set 
to  .0003  in  JITTER,  which  corresponds  to  doubling  the  radiated  power, 
relative  to  small  speeds,  when  the  speed  is  7.6  knots.  The  third  correction 
factor  is  needed  because  power  is  usually  not  radiated  equally  strongly  in  all 
directions.  The  parameter  p  must  be  such  that  I  p  |  <1.  When  P  >  0,  power 
tends  to  be  radiated  most  strongly  broadside.  P  is  arbitrarily  set  to  .5  in 
JITTER,  which  corresponds  to  broadside  radiation  being  three  times  as  strong 
as  bow/ stern  radiation.  The  submarine  is  assumed  to  be  oriented  in  the  same 
direction  as  its  velocity  vector.  The  model  sketched  above  is  correct  in  its 
tendencies  for  a  non-cavitating  submarine,  although  the  parameters  would  of 
course  need  to  be  adjusted  in  a  real  application.  A  more  detailed  description 
of  radiated  noise  can  be  found  in  Ross  [1976]. 

P(t)  as  given  by  (3)  depends  ^ n  submarine  position  x(t)  (needed  to 
compute  rj(t))  and  velocity  z(t)  (needed  to  compute  v(t)  and  0j(t)),  so  the  path 
optimization  problem  is  in  the  class  described  in  Section  2.  PTTER  uses  the 
optimization  method  described  in  the  Appendix  to  solve  it.  The  step  size  is 
dynamic;  as  long  as  the  forecast  improvement  agrees  with  the  actual 
improvement,  JITTER  will  gradually  increase  it.  If  the  forecast  and  actual 
improvements  disagree  sufficiently,  the  step  size  is  reduced.  The  current 
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route  and  the  objective  function  are  continuously  displayed,  and  the  user  is 
invited  to  stop  the  program  whenever  convergence  has  occurred,  possibly 
restarting  it  on  a  different  route  that  connects  the  same  two  points.  The 
reader  is  invited  to  try  using  JITTER,  which  is  included  on  the  attached 
diskette. 

One  of  tne  more  interesting  features  of  JITTER  solutions  is  the  presence 
of  "kinks"  (discontinuities  in  velocity)  on  the  optimal  route  in  problems 
where  the  input  average  speed  is  low  (5  knots,  say).  These  kinks  are  at  first 
surprising  but  actually  reasonable,  since: 

1)  Velocity  is  the  control  variable,  and  optimal  control  problems  often 
involve  discontinuities  in  control  variables. 

2)  The  submarine  would  like  to  avoid  exposing  its  sides  to  close 
listeners  because  P  >  0.  Therefore  sharp  turns  through  significant 
angles  ana  bow-on  approaches  to  obstacles  should  be  expected. 

Real  submarines  can't  make  sharp  turns,  so  any  proferred  "solution"  with 

sharp  turns  in  it  is  of  course  only  a  rough  guide  to  what  should  actually  be 

done.  Kinks  disappear  when  the  input  average  speed  is  high  (20  kt,  say).  The 

kinking  problem  could  be  avoided  by  making  velocity  a  state  variable  and 

letting  the  control  variables  be  (say)  rudder  angle  and  acceleration. 

To  run  JITTER,  an  MS-DOS  computer  with  either  an  EGA  or  VGA 
graphics  board  is  required;  the  program  tests  for  the  right  hardware  and  will 
terminate  if  conditions  aren't  satisfactory.  The  8087  chip  is  not  required,  nor 
is  a  color  display  (but  by  all  means  choose  a  color  display  if  one  is  available). 
JITTER  first  reads  the  chart  scale  and  listener  locations  from  the  file 
SITES.DAT  that  is  included  on  the  diskette  (along  with  a  file  SITES.EXE  that 
can  be  used  for  changing  SITES.DAT  if  desired).  JITTER  echoes  this  data  to 
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the  screen,  asks  for  an  average  speed  input,  and  then  produces  a  map  on 
which  the  user  inputs  a  candidate  route  m  a  connect  the  dots  fashion.  The 
solid  disks  on  the  map  represent  the  listeners,  with  the  ->rea  of  each  disk  being 
proportional  to  the  listener's  sensitivity  (there  is  no  special  significance  if  a 
submarine  track  penetrates  one  of  these  disks).  Once  calculations  start,  the 
current  route  is  displayed  using  line  segments  of  alternating  color,  each  of 
which  corresponds  to  a  fixed  amount  of  time,  so  an  impression  of  speed  can 
be  got  from  the  lengths  of  the  segments.  Calculations  will  be  interrupted 
when  JITTER  senses  that  no  further  improvement  is  possible,  in  which  case 
the  user  can  either  terminate  the  program  or  input  another  candidate  route 
that  connects  the  same  initial  and  terminal  segments.  In  the  latter  case, 
JITTER  displays  the  smallest  measure  of  effectiveness  (received  energy)  that 
has  been  achieved  by  previous  tries. 

It  should  be  mentioned  that  decisions  aids  like  JITTER  could  deal  with 
moving  listeners,  as  long  as  the  track  that  each  listener  follows  is  known. 
The  main  complication  would  be  graphical,  rather  than  mathematical,  since 
it  would  become  difficult  to  display  the  listener  tracks  while  asking  the  user  to 
input  a  reasonable  starting  solution. 

Other  points  worth  noting  about  JITTER 

1)  The  vertical  dimension  reads  downwards!  This  is  the  author's 
revenge  on  submariners,  who  are  forever  plotting  transmission  loss 
curves  in  that  manner. 

2)  The  most  likely  reason  for  a  division  by  zero  termination  is  a  leg  of 
zero  length  The  easiest  way  to  do  this  is  to  fail  to  move  the  cursor 
between  inserts. 

3)  Don't  route  the  submarine  directly  over  one  of  the  listeners.  If  you 
do,  JITTER  will  respond  by  lengthening  the  segment  that  passes  over 


the  listener,  rather  than  avoiding  the  listener.  This  action  "works" 
because  distances  are  only  evaluated  at  the  ends  of  line  segments;  the 
phenomenon  could  be  avoided  by  increasing  the  number  of  line 
segments,  but  doing  so  would  slow  the  program  down. 

4)  The  user's  first  input  will  be  "average  speed  as  the  crow  flies,"  this 
input  being  used  only  to  calculate  the  amount  of  time  available.  The 
actual  average  speed  of  the  submarine  on  the  initial  track  will  be 
greater  than  this  to  the  extent  that  the  track  is  longer  than  the 
shortest  distance  between  A  and  B. 

5)  The  source  code  in  Turbo  Pascal  is  available  from  the  author  (408- 
646-3127). 
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APPENDIX 

Consider  the  problem  of  minimizing 

I*  f(x,dx  /  dt,t)dt  (Al) 

where  x(0)  and  x(T)  are  given,  x  and  dx/ dt  depend  on  t,  but  this  dependence 
on  time  is  suppressed  in  the  notation.  This  is  the  classical  fixed  end  point 
problem  of  the  Calculus  of  Variations.  We  first  approximate  (Al)  by  a  sum. 
Let  T  =  N5,  where  N  is  a  large  integer,  let  Xj  =  x(i5);  i  =  0,  ...,  N,  and  let 
X[  =  (xj  -  Xj_i)/8,  i  =  1,  ...,  N.  Then  the  problem  is  to  minimize 

m  =  «£/(*i,ii.i«)  (.«)' 

i=l 

by  choosing  xj, ...,  xn_!  optimally,  with  xo  and  xN  given.  The  method  described 
below  is  a  first  order  gradient  method  (Bryson  and  Ho  [1969])  wherein  Xj  is 
modified  to  x;  +  auj  and  X\  is  modified  to  X\  +  cuij,  i  =  0,  ...,  N.  a  is  a  small 
scalar  and  uo  =  un  =  0,  with  u,  being  otherwise  arbitrary  for  the  moment. 
Letting 

N 


/(a)  =  $£/(*,-  +  ccui,Xi  +  aui,i8), 

i=i 

(.«) 

fi1  =-^-f(xi/xi/i8),  and 

(214) 

II 

H- 

>» 

(215) 

(A2)  is  the  simplest  discrete  version  of  (Al),  but  it  is  unsymmetric  with 
because  i,  involves  x;_i  but  not  x,*,.  A  symmetric  version  is  actually  used 
otherwise  JITTER'S  method  is  as  described  here. 

respect  to  time 
in  Jitter,  but 

11 

we  have  that  the  derivative  of  J  with  respect  to  a,  evaluated  at  a  =  0,  is 

r  =  <s£(/iV/i2«i).  (A6) 

i=  1 

The  symbol  ii,-  is  being  used  to  symbolize  the  backward  difference 
(uj-Uj.i)/5;  with  g  below  similarly  being  the  backward  difference  of  g.  Note 
that  fj  and  ff  are  vectors  of  the  same  dimension  as  x,  and  that  the  products  in 
(A6)  are  inner  products. 

Let  gi  =  ^  fj ;  i  =  0,...,N,  so  that  f}  =  g,-.  Using  summation  by  parts, 

we  have 

N  N  N 

S'Lfi\  =  S^giUi  =  uNgN  -  U0g0  -  gi-i«r  (A7) 

j=i  1=1  i=i 

Using  (A 7)  and  the  fact  that  uq  =  ujvj  =  0,  (A6)  is 

N 

/'  =  where  (A8) 

i=l 

hi  =  f^  -gi-i-  Recall  that  the  quantities  uj  are  still  unspecified,  except  that 

w,-  =  0  because  un  -  go  =  0.  If  J'  is  to  vanish  regardless  of  how  u;  is 

specified,  it  follows  that  hj  must  be  constant  for  i  =  0,  ...,  N.  (A8)  therefore 
determines  the  optimal  trajectory  to  within  a  (vector)  constant.  Finding  the 
constant  that  is  consistent  with  the  boundary  conditions  is  a  two-point- 
boundary-value  problem,  with  erroneous  constants  corresponding  to 
trajectories  that  are  optimal  but  not  feasible.  However,  an  iterative  method 
where  all  trajectories  are  feasible  can  also  be  based  on  (A8),  since  for  any 
feasible  trajectory  the  (generally  nonconstant)  quantities  hj  determine  an 
improving  direction.  Specifically,  let  go  =  0  and 
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Ui  =  h-hj ;  i  =  where 


(A9) 


—  1  —  - 

/i  =  ■  Subtracting  each  hj  from  the  average  h  forces  un  to  be  0,  as 

required.  Substituting  (A9)  into  (A8),  we  have 


N 


/'  =  S^hj(h  -  =  SNQ,  where 

i=l 


(A10) 


Since  Q<0,  with  equality  possible  only  when  hj  is 


constant  (that  is,  hj  is  independent  of  i),  (A10)  results  in  a  negative  value  for  ]' 
unless  the  trajectory  is  already  locally  optimal. 

An  all  feasible  gradient  method  is  now  clear: 

1)  guess  a  feasible  trajectory  x,;  i  =  0, ...,  N 

2)  let  xt  =  (xi  -xi_1)/  S;i  =  1 

3)  use  (A9)  to  determine  li, ; f  =  1,...,N 

4)  let  Ui  =^'=1w;;i  =  l,...,N-l 


5)  let  Xi  =  Xj  +  auf,i  = 


6)  replace  xj  with  Xj  and  go  to  2). 

The  only  remaining  issues  are  the  usual  ones  in  first  order  methods: 
determination  of  the  step  size  a  and  the  stopping  criterion.  Since  aj'  is  a 
forecast  of  the  decrease  to  be  expected  with  each  modification,  a  natural 
dynamic  step  size  method  would  increase  (decrease)  a  if  the  forecast 
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improvement  is  (is  not)  approximately  equal  to  the  actual  improvement. 
This  is  the  method  used  in  JITTER. 
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